Failure Detection and Partial Redundancy in Hpc

KHARBAS, KISHOR H. Failure Detection and Partial Redundancy in HPC. (Under the direction of Dr. Frank Mueller.) To support the ever increasing demand of scientific computations, today’s High Performance Computing (HPC) systems have large numbers of computing elements running in parallel. Petascale computers, which are capable of reaching a performance in excess of one PetaFLOPS (1015 floating point operations per second), are successfully deployed and used at a number of places. Exascale computers with one thousand times the scale and computing power are projected to become available in less than 10 years. Reliability is one of the major challenges faced by exascale computing. With hundreds of thousands of cores, the mean time to failure is measured in minutes or hours instead of days or months. Failures are bound to happen during execution of HPC applications. Current fault recovery techniques focus on reactive ways to mitigate faults. Central to any kind of fault recovery method is the challenge of detecting faults and propagating this knowledge. The first half of this thesis work contributes to fault detection capabilities at the MPI-level. We propose two principle types of fault detection mechanisms: the first one uses periodic liveness checks while the second one makes on-demand liveness checks. These two techniques are experimentally compared for the overhead imposed on MPI applications. Checkpoint and restart (CR) recovery is one of the fault recovery methods which is used to reduce the amount of computation that could be lost. The execution state of the application is saved to a stable storage so that after a failure, computation can be restarted from a previous checkpoint rather than from the start of the application. Apart from storage overheads, CRbased fault recovery comes at an additional cost in terms of application performance because normal execution is disrupted when checkpoints are taken. Studies have shown that applications running at a large scale spend more than 50% of their total time saving checkpoints, restarting and redoing lost work. Redundancy is another fault tolerance technique, which employs redundant processes performing the same task. If a process fails, a replica of it can take over its execution. Thus, having redundant copies decreases the overall failure rate. The downside of this approach is that extra resources are used and there is an additional overhead of performing redundant communication and synchronization. The objective of the second half of the work is to model and analyze the benefit of using checkpoint/restart in coordination with redundancy at different degrees to minimize the total time and resources required for HPC applications to complete. c © Copyright 2011 by Kishor H. Kharbas